A Roblox Case Study

Current State

Currently, the Roblox transaction page presents monetary incentives (the cost of Robux per dollar decreases the more you spend). More recently, a promotional item is presented as only available when purchasing either of the largest two bundles (figure below).

Monetary Incentives

The current monetary incentives for subscription purchases work well for those that understand their robux spend per month (e.g. allowances for kids, or budget minded teens/adults, creators/brands that spend robux consistently, etc.). While in contrast, the monetary incentives to purchase a larger bundle work well for impulse buyers that understand how/where they may immediately spend it.

Exclusivity Incentives

More effective however are incentives focused on exclusive items or exclusive access. The Gold Clockwork Headphones offered for a limited time only is an attempt to introduce an exclusivity incentive, however, many users have noted on popular Roblox community forums that these headphones have been available for so long that the perception of exclusivity is possibly dampened. Additionally, what’s not presented and would likely be a larger effort change, would be exclusive experiences only available to those that subscribe to Robux purchases.

A Comparible Example

Poker Stars VR is a very popular game among players on the Quest VR platform and requires players to purchase In-Game currency (poker chips) with real dollars.

• Poker Stars VR also provides users with the option of subscribing to monthly re-occurring purchases of their in-game currency.
• In addition to playing poker, in-game currency (chips) can be used to personalize a player’s avatar or purchase items that are entertaining to play with in VR while sitting at a poker table.

Poker Stars VR does not provide monetary incentives to purchase more poker chips, instead, the game emphasizes exclusivity incentives.

• Subscribers of the in-game currency are given access to an exclusive ‘Member’s Lounge’ where players can hang out and socialize with other members while not actively playing at a poker table.
• Furthermore, subscribers are given items that are only available to them and receive a badge next to their usernames that identifies them as “Members” to all other players in the game. These users will play with their exclusive items at poker tables (which based on personal experience, are incredibly fun to play with), and often motivate a non-member sitting at the same table to become a member and have access to the same items.

Proposed Change

In the case of the Roblox Gold Clockwork Headphones, a more effective approach would be the following recommendations:

Count Down Clock

As mentioned earlier, many users have noted on popular Roblox community forums that these headphones have been available for so long that the perception of exclusivity is possibly dampened.

Therefore, when displaying an exclusive item, include a count down clock whereby the item is no longer available when the timer ends. This would have the intended effect of “Limited Time Only!” while instilling a sense of urgency to the user’s purchase decision.

Mystery Boxes

Instead of offering specific items, offer mystery boxes with unknown exclusive gifts inside.

• This is a popular approach in the real-world for consumers of “drop culture” that spend hundreds of dollars on a box of collectibles where the only information known to the purchaser is that once the box sells out, it can never be purchased again.
• Also similar in concept are collectibles like Pokémon cards where the idea that a bundle of cards may in fact contain a highly coveted item is enough to motivate larger quantity purchases.

The following figure illustrates what the transactions page may look like after implementing these recommendations:

AB Testing

This section illustrates how I would go about simply just setting up an AB test to measure the impact of the changes I proposed above.

Define Success Metric

Since we are targeting a change designed to incentivize purchases of higher quantity Robux bundles, the success metric I’ll be choosing is the average Robux purchase size per user.

This metric meets the following guidelines:

• Measurability – (assumption) we have the data on users that purchase Mystery Box Robux bundles against their own average historical order size
• Attributable – this metric is attributable (we can assign cause to effect) by making the following comparisons:
  • Users who purchase Mystery Box bundles vs. their own historical purchase behavior.
  • Users who are presented a Mystery Box incentive vs users who are not presented this incentive (in the same time period).
• Variability – We can be confident this metric has low variability and effectively distinguishes between treatment and control because of the following:
  • (Assumption) Typically users with repeat and regular purchases gravitate towards the same bundle each time they need to purchase more Robux. Deviations from their mean may occur due to external factors influencing their purchase such as changing economic conditions, however, over a long enough historical period, there is strong deviation to the mean.
  • Roblox offers only 5 bundles for purchase, and more important, the quantities of Robux in each substantially increase (by 2x or more) across bundles. The limited choices presented further ensures low variability in historical purchases.
• Timeline – Based on what is publicly known about Daily Active User counts on the platform, we can assume transactions occur at a high enough volume and velocity to enable us to keep the experimentation timeline as short as 1 week.

Define Secondary Impact

A secondary impact to be aware of is the frequency of purchases. If we incentivize the purchase of larger bundles, but as a result, this decreases the overall frequency of Robux purchases per-user, then we may end up with no overall revenue impact to the company from launching this change. I hypothesize that larger bundle purchases lead to more time spent on the platform, and the frequency of user purchases will remain constant for users incentivized by the Mystery Box.

Setup Hypothesis Testing

Next let’s set up our hypothesis testing by stating the following:

• Null Hypothesis: the average Robux purchase size per-user remains unchanged between the group that is presented the mystery box and the group that isn’t.
• Alternative Hypothesis: The average Robux purchase size per-user is different across these groups.

We’ll use the standard thresholds for significance level (p-value) and statistical power (r-squared) of .05 and 80%.

Additionally, we’ll require a Minimum Detectable Effect (MDE) of 1%, which is the standard for companies the size of Roblox.

Design The Experiment

Now let’s design the experiment with the following parameters:

• Randomization Unit: User
• Target Population: Visitors of the Roblox Transactions page
• Sample Size: Simple formula using sample standard deviations for treatment and control groups
• Duration of Experiment: 1 week

Remaining Steps

The next steps require internal Roblox data which we don’t have access to. For the purposes of completeness, I’ll outline them here:

1. Run the experiment (ensuring pipelines are setup to collect resulting data)
2. Validation
   • Instrumentation Effect (validate data captured for correctness)
   • External Factors (specific to experiment time-period)
   • Selection Bias (AA Test)
   • Sample Ratio Mismatch (Chi-squared Goodness of Fit)
3. Interpret results (Can we reject the null hypothesis given our statistical thresholds?)
4. Launch Decision – ROI based decision that considers factors such as the Secondary Impact metric defined earlier, as well as the overall cost of launching Mystery-boxes widely